*7 - 206047 


/\f /•) / CR-203878 


/ / • 

// ' ~ ^ 



International Ultraviolet Explorer 
Final Archive 
Contract NAS5-97042 


Final Report 


Computer Sciences Corporation 
System Sciences Division 
10000A Aerospace Road 
Lanham-Seabrook, Maryland 20706 


Major Offices and Facilities Throughout the World 



NAS5-97042 Final Report 


Preface 

This submission is the Final Report for the International Ultraviolet Explorer (IUE) Final Archive 
contract, NAS5-97042. The term of this contract is November 6, 1996, to October 9, 1997. The 
objective of this contract was the completion of GSFC activities associated with the processing 
and archiving of IUE Final Archive data. That objective was successfully met. 
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International Ultraviolet Explorer Final Archive 
Contract NAS5-97042 


1. Work Accomplished 
a. Final Archive Processing 

CSC processed IUE images through the Final Archive Data Processing System. Raw images 
were obtained from both NDADS and the IUEGTC optical disk platters for processing on the 
Alpha cluster, and from the IUEGTC optical disk platters for DECstation processing. Input 
parameters were obtained from the IUE database. Backup tapes of data to send to VILSPA were 
routinely made on the Alpha cluster. 

IPC handled more than 263 requests for priority NEWSIPS processing during the contract. Staff 
members also answered various questions and requests for information and sent copies of IUE 
documents to requesters. 

CSC implemented new processing capabilities into the NEWSIPS processing systems as they 
became available. In addition, steps were taken to improve efficiency and throughput whenever 
possible. The node TORTE was reconfigured as the I/O server for Alpha processing in May. The 
number of Alpha nodes used for the NEWSIPS processing queue was increased to a maximum of 
six in measured fashion in order to understand the dependence of throughput on the number of 
nodes and to be able to recognize when a point of diminishing returns was reached. With Project 
approval, generation of the VD FITS files was dropped in July. This action not only saved 
processing time but, even more significantly, also reduced the archive storage media requirements, 
and the time required to perform the archiving, drastically. 

The throughput of images verified through CDIVS and processed through NEWSIPS for the 
contract period is summarized below. The number of images of a given dispersion type and 
camera that were processed in any given month reflects several factors, including the availability 
of the required NEWSIPS software system, the availability of the corresponding required 
calibrations (e g., the LWR high-dispersion ripple correction and absolute calibration), and the 
occurrence of reprocessing efforts such as that conducted to incorporate the updated SWP 
sensitivity-degradation correction in May. 
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Month 

CDIVS 

LWP 

LWR 

SWP 

LWP 

LWR 

SWP 

November 

485 

60 

0 

119 

0 

0 

3414 

December 

530 

8 

0 

56 

0 

0 

3325 

January 

270 

144 

0 

229 

0 

0 

3275 

February 

18 

172 

13 

628 

0 

0 

453 

March 

47 

0 

1786 

0 

0 

0 

15 

April 

19 

90 

2417 

15 

0 

0 

0 

May 

223 

38 

654 

1389 

0 

0 

2480 

June 

284 

0 

2830 

0 

880 

0 

5 

July 

26 

66 

2194 

617 

4813 

0 

28 

August 

284 

291 

801 

1865 

1879 

4153 

323 

September 

310 

432 

710 

1091 

393 

350 

139 

October 

67 

53 

37 

83 

81 

79 

100 

Total 

2563 

1354 

11442 

6092 

8046 

4582 

13557 

A total of 487 tapes of NEWSIPS data, processed on the Alpha cluster and the DECstations, was 

shipped to VTLSPA during the contract period. 

An accounting of the BOLS sent to the NSSDC 

as well as the tapes shipped to VTLSPA 

is contained ir. the table below. 








VILSPA 




Month 

BOLs (Alpha) BOLs (DEC) 

Tapes 




November 

23 


3 

65 




December 

23 


3 

10 




January 

22 


3 

20 




February 

5 


11 

93 




March 

1 


19 

40 




April 

0 


27 

35 




May 

10 


24 

0 




June 

6 


32 

54 




July 

24 


32 

46 




August 

24 


26 

80 




September 

8 


29 

0 




October 

6 


4 

44 




Total 

152 


213 

487 
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b. Calibration 

During the contract period, CSC completed the derivation and/or evaluation of a number of 
important calibrations for NEWSEPS. These include the following: 

• LWR trailed-to-large aperture and large-to-small aperture low-dispersion absolute calibration 
correction ratios for ITF B 

• LWR sensitivity degradation correction for ITF B 

• LWR, SWP, and LWP large- and small-aperture areas 

• LWR NEWSIPS high-dispersion spatial and spectral resolutions 

• LWR low-dispersion absolute calibrations for ITFs A and B (evaluated the calibration derived 
by VILSPA) 

• LWP high-dispersion ripple correction and absolute calibration (evaluated the calibration 
derived by VILSPA) 

• LWR high-dispersion ripple correction and absolute calibration (evaluated the calibration 
derived by VILSPA) 

The calibrations derived by CSC were all documented in the 1UE NASA Newsletter and NEWSIPS 
Information Manual, Version 2.0. In addition, the NEWSIPS manual itself, representing a major 
update of Version 1 to include LWR low-dispersion and SWP, LWP, and LWR high-dispersion 
algorithms and calibrations, was completed. It has been submitted for publication as a special 
issue of the IUE NASA Newsletter and is to be made accessible in electronic form on the World 
Wide Web. 

CSC performed a number of investigations and analyses to resolve questions about NEWSIPS 
algorithms or calibrations that arose during the contract period. The questions, issues, and 
investigations are summarized below. 

• Undesirable spectral signatures were imposed by the low-dispersion absolute calibrations 
derived by VTLSPA, especially for the LWP camera. Analysis indicated this is apparently 
related to intrinsic residual detector non-linearities and the piecewise combination of 100- 
percent and 200-percent-level exposures of standard stars to derive the inverse sensitivity 
function. VILSPA indicated they would look into possibilities for remedying this situation. 

• Inaccuracies were detected in the extrapolation of the low-dispersion sensitivity-degradation 
correction for SWP data for post- 1995 observation dates. CSC analyzed the most recent 
standard-star and white-dwarf observations available (1995 and 1996) to update the SWP 
point-source sensitivity-degradation correction. CSC also determined that the existing SWP 
trailed-source and LWP point-source and trailed-source sensitivity-degradation corrections 
were valid and did not require updates. The LWR sensitivity-degradation correction was 
never in question inasmuch as the LWR correction was derived from the closed dataset of 
LWR observations and did not involve any time extrapolation. 

• LWP high-dispersion order-registration anomalies were seen in the data set of images 
processed for derivation of the LWP ripple correction. The combination of detector 
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sensitivity and source spectral energy distributions caused problems for certain classes of 
LWP spectra. New checks for properly-bounded registration solutions were added to the 
NEWSEPS algorithms for the LW cameras to protect against potential “runaway” solutions by 
reverting to global or time/temperature-predicted registration shifts in such cases. 

• Apparent flux deficiencies in specific high-dispersion orders as extracted by NEWSIPS (three 
orders for the LWP camera, one order for the LWR camera) were seen in the ripple- 
correction data sets for the LW cameras. This was traced to an extraction-centroiding 
problem caused by an obscure numerical instability involving the interplay among the precise 
starting location of the centroiding window, the search-window size, and the extraction slit 
length. The problem was eliminated simply by adding a small (subpixel) perturbation to the 
starting search location. It was also verified that SWP spectra are not affected. 

• Analysis conducted by CSC determined there are no clear benefits in terms of photometric 
noise, reliability of background fluxes, or improvements in the wavelength scale, associated 
with the use of ITF B for LWR high-dispersion images. As a result, NEWSIPS high- 
dispersion processing for LWR data uses only LWR ITF A. This is in contrast to LWR low- 
dispersion, where clear-cut improvements result from the ability to choose between ITF A or 
ITF B on an image-by-image basis. 

• Apparent ripple-splicing and background-level anomalies were reported in certain SWP high- 
dispersion spectra of B stars by a user. CSC determined that the NEWSIPS background- 
determination module was performing according to design and that the observed behavior of 
the background as modeled by NEWSIPS reflects a limitation in the ability of an automated 
process to respond to localized background variation or the effects of peculiar pathologies in 
individual images. A detailed description of the background determination algorithm was 
written and made available to users in advance of the publication of the NEWSIPS 
Information Manual, Version 2.0, via the World Wide Web. 

• Anomalously low sigma values in some LWR low-dispersion images resulting in higher-than- 
normal fluxes as extracted by the SWET module were reported by a user. CSC determined 
the cause of this behavior to be the improper extrapolation of the low-dispersion noise model 
polynomial fits at large FN values. This issue was resolved by setting the noise value at large 
FNs to the peak of the polynomial curve representing noise as a function of FN rather than 
allowing it to follow the turnover of the fitted curve. The affected data were reprocessed and 
re-archived. The SWP and LWP low-dispersion noise models were then also examined and 
found to exhibit this same extrapolation behavior, but to a much, much smaller degree since 
the turnover of the fitted functions for those cameras occurs at considerably higher FN levels 
than for LWR, and only at the shortest wavelengths. No remedial action was required for 
SWP and LWP. 

• CSC determined that the LWR noise-model anomaly did not affect in any significant way the 
derivation of the LWR low-dispersion absolute calibrations for either ITF A or ITF B. 
Furthermore, it demonstrated that high-dispersion data, in any camera, are not subject to this 
sort of noise-model anomaly, since in high-dispersion the noise models are handled differently, 
intrinsically limiting the noise values at high (and low) FNs. In addition, the noise values in 
high dispersion are not used by the boxcar extraction method as SWET does for low 
dispersion and thus cannot affect the extracted fluxes. 
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• CSC verified the validity of the technique used to establish the IUE wavelength calibration for 
large-aperture spectra in high dispersion. In NEWSIPS, as in IUESIPS, the large-aperture 
wavelength calibration is obtained from the small-aperture calibration by a simple zero-point 
shift corresponding to the physical offset between the apertures. Echelle grating theory as 
applied to the IUE instrument design led the calibration staff to question whether a small 
wavelength-dependent dispersion differential might be detectable between small-aperture 
high-dispersion data and large-aperture high-dispersion data calibrated in the standard fashion. 
Tests performed by the staff on both IUESIPS and NEWSIPS data sets, however, showed 
that small- vs. large-aperture wavelength differences, obtained by cross-correlating small- and 
large-aperture extracted spectra of the same target, exhibited only random offsets that could 
not be distinguished from target-centering errors in the large aperture. We therefore 
concluded that the IUE large-aperture wavelength calibration methodology is adequate as 
implemented. 

• In support of VTLSPA’s compatibility testing of the NEWSIPS high-dispersion Ultrix system 
against the corresponding Open VMS version, and to better understand the quantitative 
differences between the results obtained on the two systems, the calibration and software 
staffs worked together to conduct a series of tests using the images SWP2860 and SWP5136. 
Investigations made last fall indicated that differences in the final net fluxes obtained with the 
two systems were ultimately traceable to minute irreducible differences (hundredths or 
thousandths of a pixel) in the displacements in the ORD.MSK and VD.MSK files, caused by 
differences in machine architecture and calculational precision. The new tests reconfirmed this 
result and demonstrated that the Ultrix software is functioning correctly. Final Ultrix fluxes in 
high dispersion can vary from the Open VMS fluxes by amounts depending on the flux levels, 
with the largest percentage differences in cases where the background level is very low and/or 
where there is essentially no gross flux. 

• An error in the way the sensitivity-degradation correction was applied for LWP and LWR 
high-dispersion data with NEWSIPS 3.3.1 and 3.3.2 was uncovered by the IUEDAC. 

Analysis showed that a hardcoded parameter value pertinent to the SWP camera was 
inadvertently being used for both LW cameras, causing the wrong degradation correction to 
be applied for wavelengths above 2712 Angstroms. An IDL-based correction procedure was 
developed, tested, and implemented to allow the re-archiving of corrected MXHI files for the 
affected LW images. 

• The calibration staff evaluated the quality of the LWR ripple correction and absolute 
calibration derived by VILSPA. In brief, all of the images evaluated show overcorrections in 
the wings of the ripple-corrected orders at short wavelengths (typically below -2300 
Angstroms). The severity of the overcorrections is greatest for images obtained during early 
epochs, with bad focus, and through the small aperture. In general, in these same images the 
overcorrection persists to longer wavelengths than in images that are less severely 
overcorrected. The general property of overcorrection at short wavelengths was previously 
also found for LWP images, although not as extremely so. The overall quality of the 
NEWSIPS ripple corrections appears poorer at short wavelengths, comparable at intermediate 
wavelengths, and better at long wavelengths as compared to the IUESIPS results, which tend 
to show mediocre ripple corrections at all wavelengths, and in general, an undercorrection. In 
August, VILSPA declared the LWR ripple correction unacceptable and is currently exploring 
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means of improving it and re-archiving the GSFC data processed using their original 
(unacceptable) ripple correction. 

The software group developed, tested and delivered a number of NEWSEPS software releases to 
VILSPA and to the IPC during the contract year. The table below lists these deliveries in 
chronological order and summarizes their content. 


Release 

Platform 

Description 

2.5.1 

Ultrix 

3 -camera low-dispersion system, with ITF selector for LWR; 
no LWR absolute calibration for ITF B. 

3.1 

Ultrix 

SWP high-dispersion system 

3.1 A 

Open VMS 

SWP high-dispersion system 

3.2 

Ultrix 

SWP/LWP high-dispersion system, but with no LWP ripple 
and absolute calibration 

3.2_A 

OpenVMS 

SWP/LWP high-dispersion system, but with no LWP ripple 
and absolute calibration 

2.5.2 

Ultrix 

3 -camera low-dispersion system complete with all calibrations 

2.5.3 

Ultrix 

Adds updated SWP point-source sensitivity-degradation 
correction and correction for LWR low-dispersion noise-model 
anomaly to 2.5.2. 

3.3_A 

OpenVMS 

3-camera high-dispersion system, but with no LWP or LWR 
ripple correction and absolute calibration. 

3.1.1_A 

OpenVMS 

Adds updated SWP point-source sensitivity-degradation 
correction to 3.1_A 

2.5.4 

Ultrix 

Adds DMU anomaly handling to 2.5.3 

3.3. 1_A 

OpenVMS 

Adds LWP ripple correction and absolute calibration, DMU 
anomaly handling, and updated SWP point-source sensitivity- 
degradation correction to 3.3_A 

3.3.2_A 

OpenVMS 

Adds LWR ripple correction and absolute calibration, VD-file 
suppression, and several other minor changes to 3.3. 1_A. 

3.3.1 

Ultrix 

Adds LWP ripple correction and absolute calibration, DMU 
anomaly handling, and updated SWP point-source sensitivity- 
degradation correction to 3.3. 

3.3.2 

Ultrix 

Adds LWR ripple correction and absolute calibration, VD-file 
suppression, and several other minor changes to 3.3. 1 

3.3.3 

Ultrix 

Adds corrected application of LWR/LWP sensitivity- 
degradation correction and several other minor changes to 
3.3.2 

Adds corrected application of LWR/LWP sensitivity- 
degradation correction and several other minor changes to 
3.3.2_A. 

3.3.3_A 

OpenVMS 

2.5.4_A 

OpenVMS 

Port of 2.5.4 (final 3-camera low-dispersion system) to 
OpenVMS 
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CSC also completed and delivered a major document, International Ultraviolet Explorer New 
Spectral Image Processing System (NEWS1PS Pipeline) Low and High Dispersion Functional 
Specifications and Detailed System Design. This document contains the information needed by 
users to build and execute the NEWSEPS software in either the Ultrix or Open VMS 
environments. 

c. Data Archiving Support 

IUE archive support services were provided using the NDADS systems. In all, during the 
contract period 350 electronic data deliveries, encompassing 216,638 files for 45,160 images, 
were received from the GSFC Final Archive Processing systems. All of those deliveries will have 
been archived onto optical disk platter by the end of the contract period. In addition, 99 tapes of 
NEWSIPS data from VTLSPA were received. The most recent delivery of VILSPA tapes was 
not received in sufficient time to be completed before the end of the contract period. Seven of the 
tapes have been archived. The remainder will be archived to optical disk platter on a new 
contract. 

Numerous problems with the NDADS hardware system were experienced. These problems 
include parity errors and the media going off-line during platter writes as well as frequent 
downtme for jukebox maintenance and inventories. 

Task members initiated an inventory of master tapes containing IUESIPS data. All 538 "C" 
copies of the VTL tapes (IUESIPS data from VILSPA) have been inventoried. Of the 538 tapes, 
70 (13 percent) had some type of problem. 

Eight hundred seventy-three "C" copies of 1,447 "NAS" tapes have been inventoried. Work to 
inventory the remaining "C" copies and to read the "D" copies for the unreadable tapes will 
continue on a new contract. 

CSC developed plans for copying the IUE Final Archive data to DLT media. In all, 16 tapes were 
completed. Copies of 5 of the tapes have been made for PPARC. In addition, work began on an 
effort to put IUESIPS data onto DLT, beginning with the raw-image files. 

During August it was discovered that a good portion of the data written to platter between June 8 
and July 27, 1997, was corrupted during the write-to-platter process. Although the exact reason 
has not been determined, the cause of the problem involved a subtle NDADS system failure 
beyond CSC's control. A clean copy of the Final Archive data for all but 400 images was 
preserved on DLT tape, and this was used to rewrite the data to platter. The 400 images not 
recoverable from DLT were reprocessed and re-archived. 

The Archive Support group coordinated efforts with the IUEDAC and calibration staffs to 
develop, test, and implement an IDL-based procedure for correcting archived LWR and LWP 
MXHI files affected by the NEWSIPS 3.3.1 and 3.3.2 error in applying the sensitivity-degradation 
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correction for wavelengths longer than about 2712 Angstroms. The procedure was run directly 
on the NDADS system and was used to re-archive correct copies of the data. 

d. IUE Data Analysis Center (IUEDAC) Operation 

During the contract period, special assistance was provided to users and remote user accounts 
were maintained as outlined in the table below. 

Users Given Remote User 

Month Special Assistance Accounts Maintained 


November 

12 

32 

December 

9 

32 

January 

15 

24 

February 

16 

26 

March 

14 

32 

April 

11 

26 

May 

19 

28 

June 

13 

31 

July 

13 

34 

August 

12 

27 

September 

18 

38 

October 

4 

38 


During the contract period the IUE electronic newsletter was distributed monthly, and routine 
backups and system maintenance were performed for nodes IUEWWW, KLUGE, VACUUM, 
and VOID. In all, 16 routines were added to the experimental library and 57 were implemented. 
Six new calibration files were implemented. Software registration forms were received from 17 
new remote sites and 3 requests for the IUEDAC software were received. The IUEDAC 
software registration form itself was updated, the Copernicus catalog was updated, the 
description of the Final Archive status flags was updated, and the IUEDAC Users’ Guide was 
updated. The IUEDAC software was updated to support high-dispersion Final Archive data, as 
resources allowed. The IUE Merged Log was updated three times. 

e. Computer Systems Operation and Maintenance 

As appropriate, systems personnel redistributed computer equipment to maximize overall 
productivity and coordinated identification of unused equipment to be excessed. Routine system 
maintenance and troubleshooting efforts continued. 

In early spring, systems management staff coordinated the efforts of the IPC staff, the software 
staff, and LASP personnel to establish a test environment at GreenTec for assessing the impact on 
MIDAS of the LASP’s upgrading its Alphas to the VMS 7.1 system. This testing was imperative 
to avoid potential disruption of the Alpha pipeline processing environment at LASP. After 
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running preliminary tests and establishing a comprehensive upgrade plan, the systems management 
staff accomplished the system upgrades needed to establish a test environment for OpenVMS 7.1. 
Operating system and compiler components were installed on a test platform (NEBULA) to 
mirror the expected configuration of the LASP Cluster, as follows: OpenVMS Operating System 
V7.1, DECnet for OpenVMS Alpha V7.1, DECwindows Motif VI. 2-4, Digital FORTRAN V7.1, 
DEC C V5.5, and Multinet V4.0. This environment allowed the software staff to successfully test 
NEWSIPS and MIDAS under the upgraded operating system, thereby verifying that the upgrade 
would not disrupt NEWSIPS pipeline operations. 

In March, systems personnel responded to significant security concerns that had been raised by a 
break-in on the GSFC web site. Upgrades and new installations of security measures were begun. 
In response to security briefings by the GSFC CNE, system management staff put in place new 
incident reporting procedures, and security upgrades and patches to the Unix systems were 
installed. 

The systems staff worked with NSI to upgrade to the new Cicso router installed at GreenTec in 
April; users reported an immediate improvement in network performance as a result of this 
upgrade. 

Additionally, systems personnel coordinated efforts with CSC project management to conduct a 
detailed inventory and create an annotated tabulation of all Government-Furnished computing 
equipment at the GreenTec site to facilitate contract close-out activities. 

f. Management and Administration 

On a weekly basis CSC project management collected quantitative measurements of progress in 
Final Archive development and processing activities and displayed the data for project and NASA 
use. A processing-statistics histogram was provided on monthly basis and a Gantt chart depicting 
progress on development and processing activities was provided weekly. The contents of the 
Gantt chart were modified as appropriate during the contract period to reflect the tracking of 
those activities that were most significant at the time of reporting. The final histogram and Gantt 
chart are shown on the following pages. All processible IUE images, with the exception of an 
agreed-upon small group of engineering and test images of no scientific value, were archived. 
Among the unprocessible images, the vast majority have problems with the raw image which 
prevent successful processing. 

CSC project management coordinated the attendance and reporting by technical staff at weekly 
IUE Final Archive project meetings with the NASA COTR. CSC management continued close 
interaction with project technical staff on a daily basis to supervise activities, assess the adherence 
to work plans and standards and the achievement of objectives and milestones, set priorities, 
maintain and balance resources, and identify and address any potential or actual problems. 
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CSC project management coordinated with the COTR to define those records, documents, and 
other materials pertaining to the IUE mission which could be disposed of and/or recycled, and 
those which should be retained. A complete set of IUE NAS A Newsletters, IUE Three- Agency 
Coordination Meeting Records, and IUE Users’ Committee Meeting Records has been preserved, 
along with a copy of unique technical documents such as IUE System Design reports, camera 
manuals, and other instrument or operational specifications that may of long-term value beyond 
the end of the project. These materials are expected to be turned over to the group continuing 
IUE archive support at the Space Telescope Science Institute (STScI) after the end of this 
contract. 

CSC project management coordinated the categorization of all Government-Furnished Equipment 
at the GreenTec facility assigned to this contract in order to facilitate the efficient and cost- 
effective disposition of that equipment at the close of the contract. This effort included the 
identification of potential users (i .e., other government contracts) of equipment not earmarked for 
return to the LASP or to be used by the IUE archive group at the STScI. CSC project 
management coordinated these issues with the LASP, the STScI, and the CSC Property 
Management Office (PMO). 

CSC project management completed its active efforts on personnel placement issues. Particular 
care was taken to ensure smooth completion of work within the budgetary framework of the 
contract as new assignments were arranged for project personnel after the end of the contract. 
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